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Formulas are derived for probability of error in the detection of binary 
FM signals received from a channel characterized by arbitrary amplitude- 
and phase-vs-frequency distortion as well as additive Gaussian noise. The 
remits depend on the signal sequence and can be presented in terms of aver- 
ages over all signal sequences or as bounds for the most and least vulnerable 
ones. Illustrative examples evaluated, include Sunde's method of suppressing 
inter symbol interference in band-limited FM. The effects of various repre- 
sentative channel filters are also analyzed. A solution is given for the problem 
of optimizing the receiving bandpass filter to minimize error probability at 
constant transmitted signal power. It is found that a performance from 3 to 
4 db poorer than that theoretically attainable from binary PM is realizable 
over a variety of filtering situations. 

I. INTRODUCTION 

This paper undertakes to refine and extend the state of knowledge 
concerning performance of FM systems for binary data transmission 
over real-life channels. The particular aim is application to facilities such 
as exist in the telephone plant. Efficient use of the available channels 
constrains the bandwidth allowed for a given signaling speed. The 
luxury of a bandwidth sufficient to permit frequency transitions without 
amplitude variations and without dependence of present waveform on 
past signal history would in general imply an unjustifiably low informa- 
tion rate for the frequency range occupied. We therefore concentrate 
our attention on the band-limited channel with its inherent distortion 
of the FM data wave. 

We assume a linear time-invariant transmission medium specified by 
its amplitude- and phase-vs-frequency functions and the statistics of 
its additive noise sources. The limiting noise environment in the tele- 
phone plant is typically nongaussian and not well defined even in a 
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statistical sense. Nevertheless, with the usual apology, we shall perform 
our analysis in terms of additive Gaussian noise. Justification of the 
relevancy is based on the following considerations : 

(a) Laboratory tests on data transmission systems are made at present 
by adding Gaussian noise and counting errors. Good performance in 
terms of low error rate as a function of signal-to-noise ratio under such 
test conditions is found to be indicative of good performance on actual 
channels. 

(b) Identification and removal of nongaussian disturbances is a 
feasible and continuing process which should eventually lead to a more 
nearly Gaussian description of the residue. 

Our measure of performance is expressed in terms of error probability 
vs the ratio of average transmitted signal power to average Gaussian 
noise power. In most of the work we assume white Gaussian noise is 
added at the receiver input. A convenient reference is then the average 
noise power in a band of frequencies having width equal to the trans- 
mitted information rate in bits per second. 

II. STATEMENT OF PROBLEM 

A block diagram of the transmission system under study is shown in 
Fig. 1. The data source emits a sequence of binary symbols which for 
full information rate are independent of each other and have equal 
probability. The analysis can be generalized without analytical incon- 
venience to assign a probability mi to one of the two binary symbols 
and 1 — mi to the other. In conventional binary notation the symbols 
are 1 and 0. It is convenient to express binary frequency modulation of 
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Fig. 1 — Binary FM transmission system. 
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an oscillator in terms of positive and negative frequency deviations. 
The combination of data source and low-pass filter is accordingly defined 
by the shaped baseband data wave train 

•(*) - i,b n g(t-nT) (1) 

n=— oo 

where 

&„ = 2a n - 1. (2) 

The values of a n represent the data sequence in binary notation. The 
probability is mi that the typical a n is unity, and 1 — mi that it is zero. 
The value of b n is -f-1 if a n is unity, and —1 if a„ is zero. The function 
g(t) represents a standard pulse emitted by the low-pass filter for a 
signal element centered at t = 0. 

Ideally, the oscillator frequency follows the baseband signal wave 
s(t). This would imply an output voltage from the FM oscillator 
specified by 



V(t) = A cos \<a t + O + M / s(X) d\ . 



(3) 



Here, A is the carrier amplitude, a> c is the frequency of the oscillator with 
no modulating signal applied, f is an arbitrary reference time, 6 is the 
phase at t = t , and n is a conversion factor relating frequency dis- 
placement to baseband signal voltage. The instantaneous frequency of 
the wave (3) is defined as the derivative of the argument of the cosine 
function. It can be written in the form co c + w, , where w,- , the deviation 
from midband, is ideally expressed by 

co,- = ps(t). (4) 

III the practical case, the transmitting bandpass filter restricts the 
frequency-modulated wave to the range of frequencies passed by the 
channel. The purpose of this filter is to prevent both waste of trans- 
mitted power in components which will not reach the receiver and 
contamination of the line at frequencies assigned to other channels. 
The result is a transformation of the voltage wave (3) to a band- 
limited form, which must depart in more or less degree from the ideal 
conditions of constant amplitude and linear relationship between 
frequency and baseband signal. The line also inserts variations in 
amplitude- and phase-vs-frequency which cause further departures 
from the ideal. For our purposes it is sufficient to combine the line 
characteristics with those of the transmitting filter into a single com- 
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posite network function determining the wave presented to the receiving 
bandpass filter. 

The receiving bandpass filter is necessary to exclude out-of-band noise 
and interference from the detector input. It also shapes the signal 
waveform and can include compensation for linear in-band distortion 
suffered in transmission. Two contradictory attributes are sought in the 
filter — a narrow band to reject noise and a wide band to supply a good 
signal wave to the detector. An opportunity for an optimum design 
thus exists and will be explored in this paper. 

The frequency detector is assumed to differentiate the phase with 
respect to time. The post-detection filter can do further noise rejection 
and shaping in the baseband range, but its only function in our present 
analysis is to separate the wave representing the frequency variation 
from the higher-frequency detection products. The slicer delivers 
positive voltage when the detected frequency is above midband and 
negative voltage when the detected frequency is below midband. The 
slicer output is sampled at appropriate instants to recover the binary 
data sequence. 

The noise-free input to the detector will be written in the form 

V r (l) = P(t) cos (u c t + d) - Q(t) sin (u e t + 0). (5) 

P(t) and Q(t) represent in-phase and quadrature signal modulation 
components respectively, which are associated with a carrier wave at 
the midband frequency co c with specified phase 0. Such a resolution can 
always be made, even though the details in actual examples may be 
burdensome. The added noise wave at the detector input is assumed to 
be Gaussian with zero mean and can likewise be written as 

v(t) = x(t) cos (u c t + 0) - y(t) sin (co c t + 0). (6) 

If v(t) represents Gaussian noise band-limited to ±2w c , x(t) and y{t) 
are also Gaussian and are band-limited to ±a c . If the spectral density 
of v(t) is w v (u), the spectral densities of x(t) and y(t) are given by 1 

W x {lji) = W y {u>) = W v (o>c + &>) + W v (u e — CO), | 03 | < W c (7) 

In general, x(t) and y(t) are dependent, with cross-spectral density 

Wxy{u) = j[Wv((Oc - O)) - W v (o) c + Oj)] (8) 

and cross-correlation function expressed in terms of R v (t), the auto- 
correlation function of v(t), by 

R xy {r) = -2R v (t) sin co c t. (9) 
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The cross correlation vanishes at r = 0, and hence the joint distribution 
of x(t), y(t) at any specified t is that of two independent Gaussian 
variables. 

We shall also require the joint distribution of x and y with their time 
derivatives x and y. The latter are Gaussian with spectral densities 

Wi(o)) = Wy(w) = Wl0 x (co). (10) 

The cross-spectral densities are 

10«i(«) = Wyy(0)) = J0)W x (i0) (11) 

Wxy(0)) = J0)xy(0>) = U)[w v (u) c + Co) — W„(u) c ~ ")] = — Wiy . (12) 

The cross correlations are 

Rxx(t) = I w xi (o))e JTU du = -I uw x (w) sin tco d<a 

J — oo J — 00 

= Rvi(r) 



(13) 



Rxiir) = -Riy(r) = f wMe jrw du> 

J— oo 

-00 

= / (i)[w,((l)c + co) — w v (u c — 0))] COS TCO d(ji. 



(14) 



The cross correlation of a; and x as well as of y and # vanish at t = 0, 
and hence at any instant x is independent of x, and y is independent of 
v/. The cross correlations of x and y, and of i and y, do not vanish in 
general, but do vanish in the special case in which 

w v (u e + w) = w v (u c — co). (15) 

This is the case of a noise spectrum which is symmetrical with respect 
to the midband and represents a reasonable objective in system design. 
Since the simplification in computational details is quite considerable 
when the condition of symmetry is imposed, and since the departures 
caused by lack of symmetry are not of primary interest, we shall assume 
henceforth that (15) is satisfied. The four variables x, x, y, and y are 
then independent and have the joint Gaussian probability density 
function 

i r 2 , 2 .2 , .2-| 

1 \ x + y x + y 



p(x, y, x, y) = exp 

■±7T ao or L 



(T.I 



Zero- lo\- 

= I w z (u) do} = 2 / w v (o) c + co) do) (17) 

J— oo •'— oo 
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a\ = I Wi(w) dot = 2 / (aW v (os c + w) do:. (18) 

J— 00 J— 00 

The noise-free detector input wave (5) can be written in the equivalent 
form 

V r (t) = R(t) cos [ad + <f>(t)] (19) 

where 

R\t) = P\t) + Q\t) (20) 

tan 0(/) = Q(t)/P(t). (21) 

The frequency detector and post-detection filter combine to deliver a 
wave proportional to the instantaneous frequency deviation from mid- 
band. Taking the constant of proportionality as unity, we write for the 
output wave 

*(t)=- arctan -^ = p>(<) - Q2(/) . (22) 

With the functional dependence on t understood, we write this equation 
in the form 

4>'(l) = 4 = (PQ ~ QP)/R*. (23) 

When the noise is added, the detected frequency is changed to 

,/ m * (/ > + .r)(Q + y) ~ (g + yKg+j) / 9 ^ 

* (0 = * = (P + *)' + (Q + y) 2 * (24) 

Assuming that the system does not make errors in the absence of 
noise, we can express the probability of error in a given sample of 
instantaneous frequency taken at the time t = nT as the probability 
that \p'(nT) is negative if <f>'(nT) is positive or the probability that 
\p'{nT) is positive if <f>'(nT) is negative. Since the system has memory, 
the values of P, Q, P, and Q at any sampling instant depend on the 
entire signal sequence. Our procedure is first to show how the error 
probability can be evaluated at any sampling instant for any sequence. 
We then calculate error rates for specific sequences and establish bounds 
for most and least vulnerable sequences. 

Since the denominators of (23) and (24) are inherently positive, the 
decisions are made entirely on the basis of the signs of the numerators. 
Therefore, we do not require the distribution function of the instan- 
taneous frequency itself. In fact if we let 

x + P = xi , x -\- P = ii 

y + Q = 2/1 , y + Q = y\ 
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we require only one value of the distribution function of the variable z 
defined by 

z = xiij! - yix x . (26) 

The error probability is fully determined in any specific case either by 
the probability that z is negative or by the probability that z is positive. 
That is, if F(z) is the distribution function of z, we only require the value 
of F(0). 

We shall derive a general expression for F(0) in terms of a single def- 
inite integral. From this integral we shall then obtain definite integrals 
representing bounds for the error probability when arbitrary binary data 
sequences are transmitted. No restrictions on range of signal-to-noise 
ratios are made. The results will be applied to special cases of practical 
interest. One is Sunde's binary FM system which avoids intersymbol in- 
terference in a finite band in the absence of noise. When noise is added 
in this system, the detected samples become dependent on past signal 
history. It has been found possible to give a complete treatment of the 
Sunde method, including optimization of the receiving filter for minimum 
probability of error with fixed average transmitted signal power. The 
other cases analyzed in detail are based on design parameters actually in 
use on FM data transmission terminals. 

III. GENERAL SOLUTION 

Our first observation is that when x\ and y\ are fixed, the variable z 
of (26) is defined by a linear operation on the two independent Gaussian 
variables X\ and 1/1 . Hence the conditional probability density function 
p(z \ %i ,Ui) of z when Xi and yi are given is Gaussian with readily 
determined parameters. We accordingly write 

p(z\xi,yi) = — 7T= ex P — Z o r • ( 27 ) 

<t-V'2t L * a J 

The mean z is the sum of the means of Xiiji and — yiXx , that is, 

z = X\ av ?/i - 7/1 av ±1 = x x Q - yiP. (28) 

The variance a is the sum of the variances of Xifa and y\i\ ; hence 

c- = (.x-i 2 + y{)o{. (29) 

The complete probability density function p(z) for z is obtained by 
averaging the conditional probability density function over x% and yi . 
This is done by multiplying (27) by the joint probability density 
function of X\ and y t and then integrating over all Xi and y x . Calling the 
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latter function q(xi , iji), we can express its value by substituting the 
values of x and y from (25) in (16) and integrating out the x and y 
terms. The result is 



, v i r fa - p ) 2 + fo - Q)2 ~\ 

q(xi , yd = - — exp — . 

Zt<to L ^o J 



(30) 



Then 



/oo ,.00 
/ p(g I xi , yi)q(xi , yd dxj dyi . (31) 

CO •'—00 



The probability of error when the noise-free sample of frequency 
deviation is positive is 

P+ = f v(*)dz = [ p(-z)dz. (32) 

J— oo •'0 

Likewise, when the noise-free sample is negative, we obtain a probability 
of error 

P- = [ p(z) dz. (33) 

Jo 

The problem is thus reduced to the evaluation of the triple integral 
obtained by combining (27), (30), and (31) with either (32) or (33). 
It is shown in Appendix A that the result of these operations can be 
expressed in the following form 

P + =ierfc /i( 



2 ^ (34) 

// r> / >.>-■ ./\ i.> : ! i ,.-■ ,-■ /.. .. 

2ero 



vr„ L exp {-<&) erfc — v^ — dx - 



The value of P_ is obtained by subtracting the right-hand member of 
(34) from unity. We note that <j> is positive for P + and negative for P_ . 
The symbol R is used for dR/dt where R is given by (20). In a pure 
FM wave, R = 0, but this condition cannot be maintained in a finite 
bandwidth. 

Differentiating partially with respect to R and rearranging, we obtain 

dP + _ r r^ v_rv _ flvu - x) + ay ] 

dR ' irwJo xexp \_ 2<r 2 2<n 2 J 

(35) 



smh mx(1 r x2) *dx. 



or 
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We note that 6P + /dR vanishes when R = and at no other value of R. 
The latter follows from the fact that the integrand of (35) cannot 
change sign in the interval of integration. We also find that d P + /dR 
is positive when R = 0. We conclude that P + is minimum with respect 
to R when and only when R = 0. A lower boimd on the probability of 
error for any fixed R and 4> is therefore obtained by setting R = 0, 
giving 

Pl = 9 erfc ~~aT~ 

2 V2<70 



+ ; 






Also, since P + must be monotonic increasing with \ R \, the largest 
probability of error for any fixed R and <j> occurs when R has its largest 
possible absolute value. These deductions are of aid in selecting the data 
sequences which have most and least probabilities of error. 

It is shown in Appendix A that Pi can be written in the equivalent 
form 



i r /2 

Pi = - / exp 

T Jo 



fiV/(2«n 8 ) 






M. (37) 



It is also shown that when 4> < (<ri/<r ), the limiting form for large 
signal-to-noise ratio — i.e., R large compared with o- — is given by 

r'-^M-'Y -(-£)■ (38) 

When 4> > (ai/a Q ), the limiting form becomes 
When 4> = <t\/(tq , we have the exact result 

p '-§«p(-£)- (40) 

The general equation for error probability (34) can conveniently be 
expressed in terms of the following three parameters 

■a 



2ao- 
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a 2 = SL^ (42) 



b 2 =f- 2 . (43) 

Equation (34) then becomes 

P + = i erfc p + rA= /" e _p2x2 erfc [a P (l - x) 1 - bx] dx. (44) 

Evaluation of this equation in terms of the three parameters p, a, and 6 
gives the error probability for any of the FM systems considered. 

IV. ERROR PROBABILITY VS SIGNAL-TO-NOISE RATIO 

In analog systems the performance is often expressed in terms of 
signal-to-noise ratio in the receiver output. In the case of audio and 
video signals, where subjective judgments determine the requirements, 
the signal-to-noise ratio furnishes a good criterion. In the case of data 
signals, however, performance is judged in terms of errors made, and 
the errors cannot be predicted from the signal- to-noise ratio alone. The 
error rate depends in general on the distribution of the noise values. 
Furthermore, in good systems the errors are rare and hence are associated 
with infrequent noise conditions. The central part of the noise distribu- 
tion is of less importance than the tails. 

We illustrate the difference between a straight signal-to-noise ratio 
analysis and a direct error probability calculation in FM by a simple 
example. Consider the case of a long sequence of mark signals leading to 
a constant signal frequency w c + co d . The signal wave can then be 
written in the form 

V(t) = A cos (oj c + wd)t 

( 45 ) 
= A cos oidi cos u c t — A sin osat sin (a c t. 

Comparing with (5) and noting that we are omitting the arbitrary 
phase angle 6, which is of trivial interest, we make the identifications 

P(t) = A cos u d t Q(t) = A sin u d t. (46) 

Then, by differentiation 

P'(t) = —u d A sin u> d t Q'(t) = u d A cos wtf. (47) 

If a sample is taken at t = 

P = A P = Q = Q = mA. (48) 



BINARY DATA FM TRANSMISSION 2397 

Then from (24) the error yp — cod in the detected frequency deviation 
because of additive Gaussian noise is 

, (A 4- x) (udA + y) - yx f * 

V = yp — Old = 7 -r— r— — 03 d . (49j 

(A + x) 2 + t/ 2 

In a signal-to-noise ratio calculation for the case in which the signal 
amplitude is usually much larger than the noise on the line, (49) would 
be written in the form 

a> d (l + x/A) + y/A + (xy - yx)/A 2 , . 

" = (1 + x/AY + (y/Ay *■ (50) 

If we then assume that A is large compared with x, y, x, and y, we retain 
only first-order terms in small quantities and construct the following 
approximate result, valid most of the time 

v W co d (l + x/A) + y/A - co d (l + 2x/A) 

(51) 
= (y - u d x)/A. 

The approximate spectral density of the frequency deviation error is 
then 

W„(U)) « [Wy + Ud 2 W x (0})]/A 2 

= 2(w 2 + a>d 2 )w v (u) c + «)/A 2 . 

The approximate mean-square value of error can now be found by 
integrating the spectral density function w„(o>) over all frequencies. 
However, we cannot obtain the probability of error from this value 
because we do not know the distribution function. A nonlinear operation 
has been performed on a Gaussian process, and the result must be non- 
gaussian. In this case Rice has shown that the central part of the 
frequency error distribution is approximately Gaussian. His argument 
does not apply to the tail. When the signal exceeds the noise most of 
the time, it is only the tails of the distribution which are important in 
determining the probability that an error is made in distinguishing 
between mark and space frequencies. 

Since there is no intersymbol interference in our example the exact 
expression for probability of error is given by (37) with R = A and 
4, — (a* . It can be seen from the limiting forms for large signal-to-noise 
ratio, (38) through (40), that the Gaussian approximation from (52) 
cannot approach the correct result. The result obtained from (52) must 
contain both the original and differentiated noise spectra in the argument 
of the exponential part of the approximation at large signal-to-noise 
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ratios. In (38) and (39) the exponential depends on either a or m but 
not both. 

As another example of the difference between inferences from signal-to- 
noise ratio and error probability, it is interesting to consider the case 
of differentially detected binary phase modulation. In this system the 
polarity of the present carrier wave is compared with the polarity one 
bit ago. The binary message is read as 1 for a phase reversal and for 
no phase change. By intuitive reasoning one could easily conclude that 
there would be a 3-db penalty relative to synchronous detection with a 
noise-free period. Certainly, in the differential case noise is added to both 
waves under comparison, and the bit interval is usually long enough to 
make the two noise wimples substantially independent of each other. 
Signal-to-noise ratio analysis supports the intuitive argument when the 
average noise power is small relative to the average signal power. A 
direct calculation of error probability, however, exposes the fallacy and 
reminds us sharply that the noise is not small compared with the signal 
when errors occur. If we focus attention on the large noise peaks which 
cause error, we can see that the simultaneous combination of dis- 
turbances on both waves does not imply the same probability of disaster 
as would follow from concentration of all the noise on one wave. 

The differential binary PM problem can in fact be solved as a simple 
special case of the analysis we have developed for FM. The input wave 
to the detector can be written as 

V r (t) = [P(t) + x(t)] cos a c t - y(t) sin u> c t. (53) 

The detector operates by multiplying 7,(0 and V,(t - T), selecting 
the low-frequency components of the product, and sampling the output 
at intervals T apart. If we assume <x> c T is a multiple of 2x and identify 
quantities evaluated at t — T by the subscript d, the binary decisions 
are based on the sign of the wave 

7.(0 = (P + *)(**« + *) + VV* • ( 54 ) 

When the correct binary decision is 0, the signs of P and Pd are the 
same, and an error occurs if the sampled value V a is negative. When the 
correct binary decision is 1, the signs of P and Pd are opposite, and an 
error occurs if the sampled value of V a is positive. The two cases are 
symmetric and an analysis of either suffices. For the case of the symbol 
0, P = P d , while for the case of 1, P = -P d . 

In calculating the signal-to-noise ratio for the case of a symbol 0, 
we would write 

V. = p(p + x + Xi + Ei+my (55) 
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Then if P is large compared with x, Xd , y, and yd , we approach a con- 
dition in which the decisions are based on the sign of P + x + Xd . If 
x and Xd are independent, the sum x + x<t represents samples from random 
noise with twice as much average power as the samples of either x or Xa 
alone. This tempting argument leads to the 3-db rule. 

In a direct calculation of error probability, we recognize that the 
influence of xxd and yyd cannot be ignored at the tails of the noise 
distribution where the errors occur. In particular, if x and Xd are both 
very negative, tending to cause an error in a symbol 0, the value of xx& 
is large and positive, tending to prevent the threatened damage. 

To find the error probability, we compare (54) with (26), and note 
that we have a special case of the previous solution if we make the 
following identification 

z = V a Xi = P + x iji = P d + Xd 

(56) 
V\ = V xi = -y d . 

The remainder of the solution proceeds as before if x, y, Xd , and yd are 
independent Gaussian variables. The independence is guaranteed if the 
second-order correlation functions vanish at lag time T. One difference 
between this case and the earlier one is that the variables x, y, Xd , and ijd 
all have the same variance. This specialization can be made in the earlier 
work by setting <r = <j\ = a. By comparing with (25), we further note 
that we can now set Q = P = 0, Q = Pd = P. Hence we also have 
R = P and R = 0. Corresponding to 4> we insert the value which V a /R 2 
assumes in the absence of noise, namely <j> = P /P = 1. In terms of 
(41), (42), and (43) we then have 



P 2 

P = -?-, « 2 = 1 6 2 = 0. (57) 



2 __ 

~ 2<r 2 



Hence the answer is given by (40), namely 



- n » 



P + = P_ = |(T P . (58) 

In the ideal case, a bandwidth f is sufficient to send signals by binary 
PM at a rate jo bits per second without intersymbol interference. This 
allows for upper and lower sidebands with widths /o/2. If the spectral 
density of the noise is ^o watts/cps, it follows that a = vo/o . Then M, 
the ratio of average signal power to the average noise power in a band 
of width equal to the bit rate, is equal to the ratio of P /2 to vq(q and 
hence M = p 2 . The formula for error probability is thus found to agree 
with the one given by Lawton. Average signal power 0.9 db greater 
than the coherent case is required for an error probability of 10~ . The 
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difference in performance between the differential and purely coherent 
cases approaches zero at very high signal-to-noise ratios. 

V. SUNDE'S BAND-LIMITED FM SYSTEM WITHOUT INTERSYMBOL INTER- 
FERENCE 

E. D. Sunde 4 has described a binary FM system in which the inter- 
symbol interference in the absence of noise can be made to vanish at 
the sampling instants, even when the bandwidth is limited to an extent 
comparable with that used in AM transmission. The method is remark- 
able in that a type of result similar to that given by Nyquist 5 for AM 
systems is obtained for all sequences in spite of the nonlinear FM detec- 
tion process which invalidates the principle of superposition. The per- 
formance falls a little short of the corresponding AM case, in that some 
dependence on the message appears when noise is added. 

Fig. 2 shows a diagram of Sunde's method. The binary message is sent 
by switching between two oscillators. The difference between the oscil- 
lator frequencies must be locked to the bit rate, and the oscillators must 
be so phased that the frequency transitions are accomplished with con- 
tinuous phase. The combination of sending filter, line, and receiving 
filter modify the switched output to produce a spectrum at the input to 
the frequency detector with even symmetry about the midband and 
with Nyquist's vestigial symmetry about the marking and spacing fre- 
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Fig. 2 — Sunde's band-limited binary FM system. 
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quencies. The latter must be high enough relative to the bit rate to 
prevent appreciable lower sideband foldover. 
The output of the switch is represented by 

E(t) = | [1 - s(t)] cos [(«, - ui)t + 0.] 

(59) 

+ f U + «(01 cos [(«. + Ud )t + e m }. 

In (59) A represents the amplitude of the output and must be the same 
for each oscillator. The switching function s(t) represents the baseband 
data wave of (1). When s(t) = — 1, the first term has amplitude A 
and the second term vanishes. When s(t) = +1, the first term vanishes 
and the second has amplitude .4 . The center of the band is the frequency 
u> c and the total frequency shift is 2wd ■ For minimum bandwidth the 
angular signaling frequency a- = 2w/T must be equal to 2uij . One of the 
two phase angles 0„ and 8„, can be arbitrary, but the two angles must 
differ by 180 degrees. Under these restrictions, the value of E{t) can be 
written as 

E(t) = A sin mt sin (<a e t + 0,) - As(t) cos w d t cos (u e t + 0„). (60) 

Sunde requires that the input wave to the frequency detector can be 
written in the form 

V r (t) = A sin udt sin (uj + B r ) - As^t) cos (a e t + r ) (01) 

where 8i(t) represents the data sequence with g(t) replaced by ^i(/). 
The latter must be a pulse which gives no intersymbol interference when 
the data rate is 1/T. That is, 

■»<*) = Z jo (-)"M.« - nT) (62) 

and gi(t) assumes the value unity at / ' = and has nulls at all instants 
differing from t = by multiples of T. In mathematical notation 

gi [(m - n)T] - 8 mn (63) 

and 

Sl (mT) = (-)'"&„,. (64) 

The requirement as actually stated by Sunde differs from (61) in 
that his analysis is based on a switching function which assumes the 
values 1 and at the sampling instant rather than 1 and — 1. The two 
expressions for the requirement can be shown to be equivalent. Equation 
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(61) has the advantage that the function Si(t) has the average value 
zero for a random data sequence with equal probability of the two 
binary symbols. This fact enables an easy separation of the spectral 
density of V r (t) into line spectra contributed by the first term of (61) 
and a continuous spectral density function for the second part. 

Incidentally, it is clear from (61) that all the signal information is 
contained in the second term, and that the first term can be regarded 
as a pair of pilot tones at the marking and spacing frequencies w c ± io d . 
The sole function of these pilot tones is to enable an FM detector to 
recover the message. The information carrying part of V r (t) can equally 
well be regarded as double-sideband suppressed-carrier binary AM or 
binary phase modulation, with the carrier frequency placed at a e . The 
ideal way of detecting such signals is by multiplication with a coherent 
carrier wave, which must be transmitted as part of the data wave in 
some way. Detection of V T {t) as FM has a practical advantage in that 
there is no carrier recovery problem; the wave is ready for the frequency 
detector with no further processing. The penalty for transmitting pure 
sine waves is a waste of signal power. As will be shown quantitatively 
later, such waste results in an unfavorable comparison with more 
nearly ideal systems. 

To show that the stipulated conditions are sufficient to suppress 
intersymbol interference in the detected frequency of V r (t), we identify 
P(t) and Q(t) of (5) with the applicable terms of (61) as follows 

P(t) = -A Sl (t) (65) 

Q(t) = -A sin u d t. (66) 

We then calculate 

P'(t) = -Asi'(t) (67) 

Q'(t) = -w,,A cos co,,L (68) 

If we take frequency samples at / = mT we find that since u d T = -k 

P(mT) = (-) M+1 6« 

P'(mT) = -A Sl '(mT) 

(69) 

Q(mT) = 
Q'(mT) = (-Y" +1 co„A. 
Hence in (23), evaluated at t = mT 

4 = Q/p = Ud /b m = b m u> d . (70) 
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Fig. 3 — Nyquist's condition of vestigial symmetry. 

The value of the instantaneous frequency deviation at the with sampling 
point is, therefore, equal to u>d if s(mT) = 1 and equal to — cod if 
s(mT) = — 1. Freedom from intersymbol interference is thus obtained 
if (64) is satisfied. 

As shown by Nyquist, a sufficient condition for obtaining (64) is 
that the standard pulse Q\{t) is the impulse response of a network with 
transmittance Gi(u>) of the form shown in Fig. 3, described mathe- 
matically by 



Gx(± 



LOd 



X) + <7i(±a> d + X) = 2G l (a> d ) = T < X < co,/ 



7D 



We say that a function satisfying (71) has vestigial symmetry about 
frequency u>j because it has the type of symmetry called for in a vestigial 
sideband filter with the carrier at u d . We can think of the response at a 
frequency exceeding uid by an amount X as exactly compensating the 
deficiency in the response at the frequency less than u d by the same 
amount X. The ideal low-pass filter is a limiting special case occurring 
when the transmittance vanishes for \ a \ > u) d • The amplitude can be 
associated with linear phase shift, which changes only the origin of time. 
Unnecessary complication is avoided by carrying through the calcula- 
tions with zero phase shift. 

The conditions imposed on the filters and line to transform (60) to 
(61) can be expressed in terms of the Fourier transforms of g{t) cos ujt 
and g\{t), which we represent respectively by C(o>) and G'i(oj). Both 
C(to) and Gi(u)) are purely real and are given by 



C(u) = I g(t) cos oidl cos <at dt 

= [G(u - Ud ) + G(u> + o, d )]/2 



(72) 
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FiR. 4 — Spectrum at input to detector in Sunde's FM system. 



Gi(w) = I gi(t) COS ut dt. 

J —CO 



(73) 



The result, obtained by multiplying cos (taj, + 0.) by g(t) cos o> d t or 
flfi(0» i- s to P ,ace upper and lower sidebands on the frequencies ±w c , as 
shown in Fig. 4, with spectra equal to C(u — w )/2 and (?i(w — w c )/2 
respectively on ui c . The required transmittance function for the com- 
bination of sending filter, line, and receiving filter is then 



F(«) = 



(zi(cO — COc) 



(74) 



C(w — OJ c ) 

This function transforms the second term of (60) to the second term of 
(61). It is also necessary for the first term of (60) to remain unchanged. 
The first term can be written as the difference of sine waves of fre- 
quencies w c — m and o) c + u d . These components will be unchanged by 
the operation Y(a) if 

C(±u d ) = t?i(±« d ) or Y{u c ± a d ) = 1. (75) 

It can readily be seen that the condition (71 ) required on Gi(w) translates 
to the same condition for Gi(u) where u = a — u e • 

The relations can be made clearer by working out an example. Suppose 
the switching is rectangular and there is no lost time between contacts. 
The function g(t) is then defined by 

1, -T/2 < t < T/2 
g(t) = (76) 

0, | 1 1 > T/2. 

Let the received signal 7,(0 have a full raised cosine spectrum centered 
at o) e , with vestigial symmetry about oj c + o> rf and u e — u d • We then 
write 
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Gi(u) = 







u | S 2co<i 
u\ >2w d . 



We calculate 

nt \ o [ Tl2 i ,a 2co d cos (o.r/2) 
C(w) = 2 / cos udt cos w/ aJ = - - 



F(co) = 



ir(m — w 2 )(l + cos J— J 



co<r — <o" 



U = 0) r , — W. 



(77) 



(78) 



(79) 



A 2 ITU 
4c0d COS - 



This function satisfies the required condition that Y(o) c ± w<j) = 1. 

In practice it is difficult to control two oscillators with the necessary 
precision to meet Sunde's requirements. One method of realizing the 
system approximately is to begin with two high-frequency crystal- 
controlled oscillators of frequencies w(oj c — u d ) and n(w e + o> d ), where 
n is a large integer. The phases of the two oscillators are not under 
control and are assumed to be X and 2 , respectively. Frequency step- 
down circuits are introduced after each oscillator to give outputs of 
frequency a> c — ua and co c + u d with respective phases d\/n and 62/71. 
By multiplying these two outputs and selecting the low-frequency 
component as shown in Fig. 5, we obtain a wave of frequency 2co d and 
phase (0 2 — Q\)/n. This wave can be used to control the timing of the 
binary input symbols. For the switched marking and spacing frequency 
sources we use the stepped-down component of frequency to,. — o>d 
directly and the component of frequency oj c + oj^ with reversed 
polarity. The required frequency and phase relations are then satisfied 
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Fig. 5 — Practical realization of Sunde's system. 
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except for a slow drift in the time scale caused by the lack of perfect 
stability in the original oscillators. 

To calculate the probability of error when Gaussian noise is added to 
Sunde's FM signal, we identify the values of P(mT), P'(mT), Q{mT), 
and Q'(mT) of (69) with P, P, Q, and Q respectively. The general ex- 
pression for the probability of error, (34), is expressed in terms of R 
and R. We calculate 

R = (P 2 + Q 2 )* = A (80) 

R'(0 = % \P\t) + Q 2 (t)] h 
at 

= [P(t)P'(t) + Q(t)Q'(t)}/R(t) (81) 

R = R'(mT) = (PP + QQ)/R = {-) m+l Ab mSl '(mT). (82) 
From (62) 

st'imT) = £ (-)"W[(m " n)T\. (83) 

n=— no 

From (73) we verify 



(84) 



gi (rT) = - [ Gi(co) cos (urT) du 

T Jo 

= - / Gi(m — w) cos [rT(m — co)] rfw 

7T Jo 

+ - / Gi(wrf + o>) cos [r2'(a-rf + o))] dm 

T Jo 

2 r d 

= - Gi(cod) cos rr I cos roT dw = 5 r0 . 

7T •'0 

This checks our previous requirements expressed by (63) and (64). By 
differentiating (73) and substituting t = rT, we find 

gi'(rT) = -- f " «Gi(«) sin urT du>. (85) 

T Jo 

The value of this integral in general is not zero except when r = 0. It 
appears, therefore, that at any sampling instant t = mT the value of R 
depends on all the values of b n in the sequence except b m . 

For further progress we take a specific example, namely the full 
raised cosine spectrum for Gi(co). We set 
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GiCoO = T(l + cos ^-J/2 I « I g 2c*. (86) 



Then 



ffi'(r!T) = -%-{ o/l + cos .^- ) sin corT dto 

-S7T ^u \ -w,y/ 

(87) 



r(l - 4r 2 ) 



/• ^ 0. 



From (85), we noted that r//(0) = 0. The value of R can now be found 
from (82), thus 



[m—l w / \nj 

£ + £ } n 

-6..M £(-) 



ft = (-)" ,+i b 

,„ fr m+n - b m _„ 
n(4n 2 - 1) ' 



„i^fi (m - n)[l - 4(m - n) 2 ] 

(88) 



We observe from our previous study of the integral defining the 
probability of error that for fixed R the most vulnerable sequence is 
the one which has the largest absolute value of R. The least vulnerable 
sequence is the one for which It = 0, and this can be obtained by setting 
b m + n = bm-n for all n. The maximum absolute value of R occurs when 
b m+n and bm-n have opposite signs and the signs are reversed when 11 
changes by unity. The resulting value of | R | is 

1 

K = 2/0,1 £ 



„=i n(-\n- — 1) 

(89) 

- 2/o,l (log,. 4 - 1) = 0.7726 .U. 

The upper and lower bounds for the error probability are found by 
substituting R m and respectively for R in (34). By (80) the value of 
R is constant and equal to A. From (70), <j> = b m ud . It is important to 
note that while the intersymbol interference is suppressed in the absence 
of noise the error probability with noise present does depend on the 
signal sequence. This occurs because frequency detection is a nonlinear 
process, and the effect of noise cannot be found by merely adding a 
noise wave to the detected frequency output. 

The actual spectral density of the noise facing the frequency detector 
is under the control of the system designer, since the selectivity of the 
receiving bandpass filter is not determined by the requirements thus 
far discussed. We have stated what the received signal spectrum at the 
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detector input should be, but this is a resultant of signal shaping at the 
transmitter, the transmitting filter selectivity, and the transmittance 
of the line, as well as receiving filter selectivity. The latter can be 
varied within reasonable limits if the others are adjusted in a comple- 
mentary fashion to obtain the desired output response. In evaluating 
the merit of different receiving filter designs it is reasonable to compare 
them with the same average signal power on the line. We shall also 
assume that the line has been equalized for unity gain and linear phase 
over the band so that it can be considered as a transparent link in the 
system. 

The average signal power on the line can be computed in terms of 
(a) the transmittance function Y,(u) of the receiving filter, (b) the 
required function (?i(w) representing the spectrum of the modified 
switching function gi(t) at the detector input, and (c) the statistics of 
the data sequence. Details of the calculation are given in Appendix B. 
An interesting consequence of the assumptions that the FM wave has 
continuous phase and that the frequency shift is equal to the signaling 
rate is the appearance of discrete components on the line at the marking 
and spacing frequencies even when the data sequence is random. This 
means there are transmitted sine waves which consume power but carry 
no information. An optimization procedure aimed at conserving power 
would very nearly suppress these components at the transmitter by 
balance or by sharp antiresonances and restore them to their proper 
relative amplitudes by complementary narrow-band resonance peaks in 
the response of the receiving bandpass filter. The bandwidth used to 
augment these frequencies at the receiver could in theory be made so 
small that no appreciable effect on the accepted noise would result. The 
system would then only have to deliver the average power associated 
with the continuous part of the FM spectrum. 

Actually, even a partial suppression of the steady-state components 
on the line would destroy much of the advantage of signaling by FM. 
The system would become more sensitive to gain changes and over- 
load distortion. Accurate tracking of the suppression and recovery cir- 
cuits for the marking and spacing frequencies would be difficult at best 
and would be practically impossible over a channel with carrier fre- 
quency offset. The narrow-band recovery circuits would contribute to 
a sluggish start-up time. In fact, about the only remaining resemblance 
to FM would be the use of an FM detector. If low-level tones can ac- 
tually be recovered successfully from a received wave, it would be better 
to use them for synchronous PM detection, which is a linear method 
capable of attaining ideal performance in the presence of additive Gauss- 
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ian noise. It appears that Sunde's system should carry the power in the 
steady-state components in order to deserve the name of FM. 

Standard variational procedures can be applied to find the shape of 
receiving filter selectivity which minimizes probability of error when 
the average signal power and the spectral density of added Gaussian 
noise on the line are specified. The solution of the optimization problem 
is given in Appendix B, and means are shown for completing the com- 
putation of the corresponding probabilities of error for the most and 
least vulnerable data sequences. In the case of white Gaussian noise on 
the line, the optimum receiving filter has very nearly the same cosine 
characteristic found by Sunde for optimum binary AM transmission. 
The bounds for error probability are plotted in Fig. 6 for both FM 
proper with no suppression of steady-state tones and the abnormal FM 
with marking and spacing frequencies suppressed. Also shown is the 
ideal curve representing what can be proved to give the best possible 
binary performance. The ideal curve can theoretically be obtained for 
example by coherent detection of binary phase modulation. Differen- 
tially detected phase modulation requires about 1 db more signal power 
than ideal at an error probability of 10" 4 . 

It is seen from Fig. 6 that when the suppression bands are inserted 
in Sunde's binary FM system, the theoretical performance is only about 
a half db poorer than ideal, but, as previously pointed out, this does 
not represent a true FM system. The more legitimate FM has error 
bounds from 3 to 3.5 db poorer than ideal. However, a penalty of this 
order of magnitude could be a fair trade in many cases for the advan- 
tages of a much simplified receiver relatively immune to many channel 
faults. 

VI. APPLICATION TO DATA TERMINALS FOR USE ON TELEPHONE CHAN- 
NELS 

We now apply our formulas to calculate error probabilities in binary 
FM transmission with terminals more closely resembling those actually 
in use on telephone channels. In the design of real-life terminals, the 
emphasis is placed on ruggedness and simplicity. The bit rate is not 
locked to the frequency deviation. The filters do not meet elaborate 
optimization requirements. The significant conclusion from our evalua- 
tion of error probabilities for the practical systems is that the degrada- 
tion of performance compared with the ideal is actually very slight. 

The probability of error as given in (44) is generally applicable to 
FM systems. There are three parameters, p, a, and b, given in (41) to 
(43). The first parameter p is a signal-to-noise ratio. It depends on the 
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SIGNAL-TO-NOISE RATIO IN DECIBELS 



Fig. G — Error probabilities for Sunde's binary FM system with additive 
Gaussian noise. Bounds are for most and least vulnerable sequences. Noise refer- 
ence is mean noise power in bandwidth equal to bit rate. 



ratio of instantaneous envelope of the received signal to the rms noise 
voltage at the detector input. For any given front-end filter, this pa- 
rameter can be expressed in terms of average signal-to-noise ratio at 
the input of the receiver. The parameter a depends on the ratio of in- 
stantaneous frequency displacement at the sampling time to the Gabor 
noise bandwidth, (7i/a , of the receiver. The third parameter b depends 
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on the derivative of the instantaneous envelope at the sampling time. 
For a given channel these parameters can be computed for any par- 
ticular signaling sequence. The true probability of error could conceiva- 
bly be obtained by averaging over all possible sequences, but this would 
be a formidable task. Instead we will give bounds on the probability of 
error for the most and least vulnerable sequences over a finite repre- 
sentative set of signaling intervals. 

We first consider the system in Fig. 7, which has amplitude-vs-fre- 
quency "raised cosine" type roll-off but no phase distortion. Equal filter- 
ing takes place at the transmitter and receiver. The modulator applies a 
pure FM wave of constant envelope to the transmitting filter. In other 
words, the modulator and the demodulator are ideal. The data source 
is composed of rectangular pulses. The frequency deviation in cps is 
equal to half the bit rate. These rates and deviations are characteristic 
of practical systems. 

With the aid of a digital computer, S. Habib has calculated the pa- 
rameters given in (41) to (43) for 2 10 sequences. From these calculations 
we have computed an upper and a lower bound on the probability of 
error. These results are shown in Fig. 8. The probability of error for all 
other sequences will fall between the two curves labeled "best" and 
"worst." Superimposed on the same graph is the ideal curve, which 
can only be achieved with ideal phase systems and coherent detection. 
The FM detection is, of course, incoherent. 

Our next example applies the theory to a real bandpass filter used in 
an operational data set. Fig. 9 shows the system considered. The curve 
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/■t 
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Fig. 7 — Ideal FM modulator and demodulator with transmitted and received 
signals equally shaped by "raised cosine" type roll -off amplitude characteristics 
and no phase distortion. 
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SIGNAL-TO-NOISE RATIO IN DECIBELS 



Fig. 8 — Probability of error for system depicted in Fig. 7. Noise reference is 
mean power in bandwidth equal to bit rate. 



of loss vs frequency for the filter used is given in Fig. 10. The curve 
departs from the condition of symmetry about midband, and also the 
separation between the signal and carrier bands is not sufficient to make 
overlapping effects negligible. The marking and spacing frequencies were 
assumed to be 1200 and 2200 cps, respectively, and the signaling rate 
1200 bits per second. As shown in Fig. 11, the calculated results are 
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T v(t) = cos[a> c t + \ S(i) dil 

_rL_rL_n_ 

H(J«) 



S(t) 



FM 
MODULATOR 



(SUM 



H(S) = - 



s 2 



I[ (s-sois-s*) 



FM 
DEMODULATOR 



T 
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S, = (-1.0505 ± j 2.541) 277'xlO 3 
S 2 = (-2.541 ±jl. 0505)2^" x 10 3 
S 3 = (-0.707 ± J0.707)277"X 10 3 



R = — = BIT RATE = 1200 BITS/SEC 
Af = FREQUENCY DEVIATION = 500 CPS 

Fig. 9 — Ideal modulator and demodulator with received signal shaped by filter 
characteristics used in FM data set and shown in Fig. 10. 

about 1 db better than the experimental results obtained with a random 
word generator, random noise generator, and error counter. The experi- 
mental system included an axis-crossing detector and post-detection 
low-pass filter, which do not correspond precisely with the theoretical 
model. In view of the differences cited, the agreement between calculated 
and experimental curves is good. The penalty suffered by the actual 
back-to-back channel compared with the best theoretical FM perform- 
ance is between 2 and 3 db. Somewhat more optimistic estimates have 
been given in other published studies. 78 The effects of amplitude and 
delay-versus-frequency variation in the channel are calculable by use 
of the computer programs we have established. 

It was shown in the previous sections that a lower bound on the prob- 
ability of error occurs when the parameter b is set equal to zero. For 
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Fig. 10 — Receiver bandpass filter loss vs frequency characteristic. 



2414 THE BELL SYSTEM TECHNICAL JOURNAL, SEPTEMBER 1963 



5 

2 
lO- 2 






:=:: ^ : 




























































































3 
2 








































CALCULATED 
y WORST 


























I0" 3 














































5 












calculated\ 




















SEQUENC 


E \ 


s \ 


\measured 


2 


















\ \ 


\ 






















\ 


< k N 






ICT« 


















V 


\ 


V 




















\ 


\ 


\ 




5 




















> V 


\ 






















\ \ 


\ 












ideal\ 










v 


s 




2 












. 








\ 


\ 


IL 












\ 








\ 


v 


\ 


lO" 5 












\ 








\ 


\ 


V 












\ 










v\ 


V 


5 
2 












\ 










\\ 


\ 












\ 


V 








V 


\ 














\ 








\ 




lO" 8 














\ 








\ 
















\ 










\ 
















\ 










\ 
















\ 










\ 


















\ 


























\ 




































I0" 8 



























7 8 9 10 H 12 13 

SIGNAL-TO-NOISE RATIO IN DECIBELS 



Fig. 11 — Calculated and measured error rates for system depicted in Fig. 9. 
Noise reference is mean noise power in bandwidth equal to bit rate. 

this reason we include Fig. 12, showing a set of universal curves relating 
the corresponding minimum probability of error to p and a. 

APPENDIX A 

Evaluation of Integral for Error Probability 
We evaluate the integral 

P+= dz / p(-z\x,y)q(x,y) dxdy (90) 
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Fig. 12 — General curves of minimum probability of error vs p for different 
values of a in the range of interest. 



where 



p(-z|rr,y) = 



exp 



[ 



(z + qx- p y y-\ 



«r,[2,r(rf + !/»)]» ""** L 2<r 1 2 (x 2 + ?/ 2 ) J 

/ n 1 [ (x - P) 2 + (y - Q) 2 1 
q(x,y) = 5—5 exp - * i-I-M 521 . 



(91) 
(92) 



(94) 
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The integration with respect to z can be performed at once in terms of 
the error function by substituting a new variable u denned by 

(z + Q.c - Pyf = 2aiV + »V- ( 93 ) 

The result is: 

_i_i_ r r Qx-Py 

t+ ~2 47rcr () 2 i-» i_» <n[2(.T 2 + 7/)]> 

We now transform to polar coordinates, setting 

x = r cos y = r sin dr rfy = r dr dd (95) 

We also let 

P cos + Q sin = R cos (8 - a) = R cos ^ 

Q cos - P sin = D cos (0 + /3) = D cos (^ + 7) 

where 

R* = p 2 + Q 2 = 2<roV tan a = Q/P 

D 2 = P 2 + Q 2 tan jS = P/Q (96) 

\{/ = 6 — a 7 = a + /3. 

The result of the transformation is 
1 - 2P+ = 

_£• r g ^+r) r r_ .»-»b~» 1 r * (97) 

The integration with respect to r can be performed by subtracting 
and adding the term R cos \f/ to r. This enables separation of the inte- 
grand into a perfect differential and a term which can be expressed as 
an error function. We thereby obtain 

1 r D cos(^ + 7) 

•-"- sL— w 

r — R I R 2 2 \ / fi cos A"| 

L l + V2 * 2^0 exp r w Rin v cos * v l " erf vw J * 

We note that both cos \f/ and cos (\f/ + 7) change sign when if/ is 
increased by ir and that sin 2 (^ + tt) = sin 2 1/'. Furthermore, the inte- 
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gration in (98) is over one full period in ^, and for every value of \f/ in 
the left half of the period there is a corresponding value in the right 
half at \p + 7T. Since the error function, erf z, is an odd function of z, a 
change in the sign of cos if/ or cos (^ + 7) changes the sign of the corre- 
sponding error function in the integrand. If we multiply the first term 
under the integral sign by the terms within the bracket following, we 
see that there is only one product which does not change sign at points 
it apart. The integral of the other products must vanish. The integral 
of the one which does not change sign is twice the integral over a half 
period of \f/. Hence 



R r n ( a 2 . 2 \ 

^J-w* exp rw sin v 



1 - 2P+ = 

n 1, i > (99) 

D cos(^ + 7) 



0-1 



• cos \f/ erf / - - d\p. 

From (96) and (23) 
D cos 7 = D(c0S a cos /3 — sin a sin /i) 

-*SS-K) -***-* 

D sin 7 = D(sin a cos /? + cos a sin 



Therefore 



(100) 



RD ' RD) R (101) 

1 d (r? 2 \ - dR - 1) 



D cos (1^ + 7) = D cos 7 cos ib — D sin 7 sin ^ 

. . (102) 

= R4> cos \J/ — R sin \p. 

Now substituting 3 = sin \f/ in (99) Ave rearrange to obtain 

2 2Vtt J-i V2 <Ti 

E( juation (34) of the main text is obtained from (103) by substituting 
the complementary function crfc 2=1 — erf z. 

The lower bound Pi on the probability of error for any fixed R and 4, 
was shown in the text to be obtained by setting R = 0. When this 
substitution is made in (103) and the definition of the error function 
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in terms of an integral is inserted, we obtain 

Pl =1 2 2\ e-^dx e-'dz. (104) 

2 IT Jo J o 

The parameters a and p are defined by (41) and (42). If we substitute 
p x = y the expression becomes 

Pl= l-i f e-^dydz. (105) 

2 7T Jo •'0 

The region of integration in the double integral consists of the first 
quadrant of the ellipse 

z/(a P ) 2 + vIp = 1. (106) 

After transforming to polar coordinates by setting y = r cos and 

z = r sin 0, we can perform the integration with respect to r. The result 

is 

i r T/2 T a?o 2 "1 

P, = - exp - . 2o ,2 2 fl M. (107) 

T Jo L sin + a cos 0J 

This is e<iuivalent to (37) of the main text. 

The integral has a simple value when a = 1 , which is equivalent to 
^ = ai/a . For this case the integrand is seen to become a constant 
and (40) results. This coincides with a result given for a special case by 
Montgomery. 9 By a change in the meaning of the parameters it also 
gives the error probability for differential binary phase detection as 
discussed in Section IV. In the general case, the limiting form of Pi 
for large signal-to-noise ratio can be calculated by the method of steep- 
est descents. Saddle points occur at = and = ir/2. When a > 1, 
the saddle point at = determines the asymptotic form of the in- 
tegral for large p and (38) is obtained. When a < 1, the saddle point 
at = 7r/2 is dominant and we obtain (39). 

APPENDIX B 

Optimization of Receiving Filter for Sunde's FM System 

Our problem is to find the receiving filter characteristic which mini- 
mizes the probability of error in Sunde's FM system when the average 
transmitted signal power and the spectral density of the noise on the 
line are specified. In terms of Fig. 13 the transmittance function for the 
filter is F r (w) and the output of the filter is V r (t) as defined by (61), 
(62), (71), and (73), namely 
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V r (t) = A sin mt sin (u c t + 6 r ) - Asi(t) cos (a> c t + r ) (108) 

CO 

8i(0 = E (-) n b ngi (t - nT) (109) 

C?i(± wrf - X) + d(± w d + X) = 2Gi(«,i) = T 0<X<w d (110) 

(?i(«) = / gi(t) cob at dt. (Ill) 

The input to the filter is the sum of the signal wave V(t) plus the 
Gaussian noise wave v Q (t). The wave V(t) is defined as that function 
of time which when operated on by F r (w) produces V r (t). The noise 
wave v(t) at the input to the frequency detector has a spectral density 
equal to | Y r (u) | 2 times that of v (t). 

We shall simplify our treatment by assuming a random sequence of 
data in which the two binary symbols are selected with equal proba- 
bility. The probability is then equal to 0.5 that any particular b n has 
the value +1 and also 0.5 that the value is —1. We regard V r (t) as a 
member of an ensemble of random functions with a distribution in the 
infinite number of independent random parameters b n . The randomness 
appears entirely in the function S\(t). We can calculate the ensemble 
average of Si(t) at fixed t by adding the individual averages of the terms 
in the infinite series defining Si(/). When we do this we find that the 
only random variable in each term is b„ , which assumes the values ± 1 
with equal likelihood and therefore has the average value zero. Hence 
the ensemble average of si(t), which we shall designate by (si(t)), is 
zero for any fixed value of /. It follows that si(t) can contain no periodic 
components, for the presence of any such components would give a non- 
zero average at some values of /. Therefore, the spectral density function 
of the second term in V r (t) must be a continuous function of frequency. 

To calculate the average square of si(t) over the ensemble, we note 
that Si(t) is the sum of an infinite number of independent random 
variables of form 

z n = (-)"b„gi(t - nT). (112) 

The average value of each z n is zero and the variance, or mean square 
minus the square of the mean, is equal to the square of gi(t — nT). 



uSe v(t) + v (t) 



RECEIVING 
BANDPASS FILTER 



Y r «J) 



TO 
V ftl + vftl FREQUENCY 

V r lU + Vaj DETECTOR 



Fig. 13 — Function of receiving filter in Sunde's system. 
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Since the variance of the sum of independent variables is equal to the 
sum of the variances of the individual variables, we can write 

<*■(!)>- t 9'(t-nT). (113) 

The average in (113) is an ensemble average at fixed t. We can show 
that this average is periodic in t with period T by noting that 

W(t + 2'))= I] gi\t + T-nT) 

n=— oo 

= t fli'V ~ mT ) (U4) 



m— — oo 
2, 



Therefore the average over t can be computed by averaging over a 
single period from t = to t = T. Hence the average over time which 
we shall designate by av is 

av Sl 2 (/) = I f (sfit)) *4l /V« - nT) dt 

Th f — Jo (115) 

t so r -(n-l)T 1 f°° „ 

= 7p S o'W ^=r g*W d\- 

1 n=— oo J —nT *■ " — °° 

By application of Parseval's theorem 

av Sl 2 (0 =^ f Gx\<*)d». (116) 

From (116) we deduce that the spectral density of Si(t) is given by 

. v Gl 2 (co) WdOl (w) / 11? s 

The spectral density of V r (t) can now be easily calculated. The first 
term can be expressed as the sum of sine waves of amplitude A/2 and 
frequencies w c . + o> rf and w c - m . The first term therefore contributes 
line spectra with mean square A 2 /8 at the marking and spacing fre- 
quencies. The average square of the second term can be written 

av [AV(0 cos 2 {o e t + B r )] = y av a?(t). (118) 

The spectral components comprising si(t) cos (<a c t + Or) are those of 
8i(t) shifted from their original positions to appear as sidebands around 
the frequencies ±o> c . Hence w r (w), the spectral density of V r (t) with 
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all power assigned to positive frequencies, is given by 
A 2 A 2 

ttV(tt) = -=- 6(co — co c 4- w d) + TT S(co — w c — COj) 



COrfA (?i (a) — 6) c ) 

H j-j w ^ 0. 

47T- 



(119) 



It is convenient to let w — w c = w and write for the transmittance 
of the filter 

U(u) = Y r (u> - av). (120) 

We shall also designate the spectral density of V(t) as w(u). Since 
the linear operator U(u) can be applied individually to the components 
which make up (119) we must have 

, N A 2 8(u 4- u) d ) . A' 2 8(u — o) d ) . o} d A 2 G 2 (u) hon 
8 J C ( — a>c) | 2 8|L'(ajd)| 2 47r 2 I U{u) | 2 

The average power on the line is proportional to Wo , the average 
square of V(t), which is given by 

2«d A t j£ 



f A 

Wo = / U)(U) dll = r-j- + 

J -2u d O I U{ —03d) I" 



«M *<-)!» (122) 

, codA 2 [ 2 ° d Gi\u) , 

4r" J-2a) d I U(u) | 2 

We make the reasonable assumption that | U(u) | is an even function 
of u. Combined with the further assumption that the spectral density 
of the noise on the line is symmetrical about w f , this furnishes a con- 
venient assurance of a symmetrical spectral density for the noise in the 
output of the receiving filter. Since G\{u) is also an even function of u, 
we can write (122) in the equivalent form 

A 2 , u>dA* f 2u "G 2 (u) 
n o = £ 

where 



" wr^^f'W;"" (123) 

4 A (cod) l-K' Jo A \U) 



X(n) = | U(u) | 2 . (124) 

The function X(u) is to be chosen to minimize the probability of 
error under the constraint that W is held constant. In calculating the 
optimum function, the signal power represented by the steady-state 
components can be ignored, since this power could be reduced to an 
arbitrarily small value by the use of narrow-band suppression tech- 
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niques. The constraint on the signal power is therefore that the integral 
in (123) is to be held constant. 

Let N(u) represent the spectral density of the Gaussian noise wave 
i»o(0 on the line. Then the spectral density of v(t), the noise in the output 
of the receiving filter, is X{u)N(u). In terms of the spectral density 
w„(o)) previously defined for v(t) with values symmetrically distributed 
between positive and negative frequencies, we have 

X(u)N(u) = 2w v (u + u>„). (125) 

The values of a and <n necessary to complete the calculation of the 
probability of error by (34) can now be found by substituting (125) in 
(17) and (18) giving the results 

r2ud 

= 2 / X(u)N(u) du (126) 



<T0 



a * = o r" u*X(u)N(u) du. (127) 

•'o 

If we substitute (126) and (127) into the general expression for error 
probability, (34), and attempt to formulate a variational problem, the 
expressions become unmanageable. Instead, we concentrate attention on 
the lower bound for error probability obtained by setting R = 0, (36), 
in which it is evident that to make the error probability as small as 
possible both a and oi should be made as small as possible. As shown by 
(126) and (127), <r and <n are not independent. The effect of the de- 
pendence can be taken into account by performing the minimization 
problem in two steps. First we minimize a with both <n and W held 
constant. After this solution is obtained, we find by trial the value of 
<ri which yields the lowest minimum probability of error. 

Omitting inconsequential multiplying factors, we set the variational 
problem as 



f * X(u)N(u) du + \ [ uX(u)N(u) du 

Jq •'0 



(128) 

where X and /x are Lagrange multipliers and the function under variation 
is X(u). The solution is 

X(M) " (T+W^OO' U } 
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It is straightforward to verify that this stationary value of X(u) actually 
gives a minimum value of <r and hence minimum probability of error 
for fixed values of ai and W . 

Substituting our partially optimized solution in (123), (126), and 
(127), we obtain 

ao = 2n I — - — du (130) 

Jo (14- Aw 2 )* 

n =2li L (l + Xu'U ^ (131) 



h (14- Aw 2 )* 

2 r 2u> d 



IF, 



-W t = %4~ [ 'lOi(u)\N*(u)(l + \u 2 )>du (132) 

H'. = ^M* + *H' (133) 

4p. I (ti(w,j) I 

20,,/x/, (134) 



1 

P 2 


= 


2(To" 
A 2 " : 


1 




2(ri 2 



(135) 



2I,h 



where 



ay AW ir 2 w,i(W> - IT.) 

• 2 " d | (AU) | A/>U) 
> (1 + Am 2 )* 

/ 2 = /* "' |Gi(w) |JV*(m)(1 4- Xw 2 )*dii (137) 

Ja 



Ja (14- Am-) j 



and 



73 " Jo (1 4- Am 2 )* dM * (138) 

These equations furnish a straightforward procedure for calculating 
the optimum filter characteristic. Each assumed value of A determines 
a pair of values p and ap from which the corresponding upper and lower 
bounds for the error probability can be evaluated by computer tech- 
niques. By successive trials the best value of A can be approximated to 
any desired degree and substituted in (129) to obtain the best filter 
selectivity function. In actual examples tried, this procedure could be 
shortened because the error probability turned out to be very much 
more sensitive to the value of p than to the value of ap. If this were 
known beforehand, we would place no constraint on a x in the minimiza- 



2424 THE BELL SYSTEM TECHNICAL JOURNAL, SEPTEMBER 1963 

tion of (To . This is equivalent to setting X = 0, leading to the simpler 
formulas 



1 2u d 



p 2 w 2 (W - W a ) 

['"" ' u 2 \Gi(u)\N\u) du 



\f"\Gx(u)\N\u) dul (139) 



■\ \GM\N\u) du. 

By applying Schwarz' inequality to the products of integrals in (134) 
and (135), we verify that the case of X = gives the maximum value of 
p, but that the maximum value of ap occurs when X = <» . It seems 
therefore that an intermediate nonzero value of X would be best, but 
in the cases computed the improvement obtainable in this way turned 
out to be negligibly small. 

As an example, consider the raised cosine signal spectrum in which 
Gi(u) is given by (77). We also assume a white noise spectrum in which 
N(u) is equal to a constant N . It is convenient to introduce as a signal- 
to-noise ratio the quantity M defined by 

M= ^ = £-. (141) 

This is the ratio of average transmitted signal power to the average 
noise power in a band of frequencies of width equal to the bit rate. 
Computer results show that the case of X = is practically indis- 
tinguishable from the optimum X. Hence we set X = and calculate 
for the optimum filter 

*<«>-*<«> -050' "Si H<w (142) 

This is the same cosine filter characteristic found by Sunde to be optimum 
for binary AM with synchronous detection. From (132) and (133) we 
find that with X = 

Wo _ Ws m ^l = Wt , (143) 

Hence 

W 3 = Wo/2 and W - W. = W /2. (144) 

From (139), (140), and (141) we then calculate 
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TT'o M 



4ojrfiVn 2 

a~p~ = 



(145) 



:37T 2 TF 



16u) d No(n* - 6) 

If the steady-state components were suppressed, we would set W, = 
and would then obtain p" = M, a~p~ = 1.913.1/. This would correspond 
to a 3-db shift in the direction of lower signal-to-noise ratio when the 
error probability curves are plotted against 10 login J\I '. 

The curves of Fig. G, showing the upper and lower bounds for error 
probability when Sunde's FM system is optimized, were calculated by 
S. Habib on the digital computer. The case of a nonoptimum receiving 
filter is illustrated by the corresponding curves for a rectangular baud 
defined by 

X(u) = X \u\ < 2co rf . (147) 

For this case we compute from (126) and (127) 

/•2w d 

(ro 2 = 2 / X N du = 4mX N (148) 

•"0 

(7i = 2 / uXoNodu = . (149) 

Jo o 

From (123) 

• 2 /•2u,/ / \2 P .2 



We then calculate 



4,Yn 8« fl 



p 2 = 2M/5 aV = 3M/10. (151) 

If tlie steady-state components are suppressed, the average transmitted 
power could be reduced to (§ — |)/(f) = § of the previously deter- 
mined value, which is a saving of 2.2 db. 

REFERENCES 

1. Bennett, W. R., Electrical Noise, McGraw-Hill Book Co., Inc., New York, 

I960, pp. 234-238. 

2. Rice, S. ()., Statistical Properties of a Sine Wave Plus Random Noise, B.S.T.J., 

27, Jan., 1948, pp. 109-157. 

3. Lawton, J. G., Theoretical Error Rates of "Differentially Coherent" Binary 



2426 THE BELL SYSTEM TECHNICAL JOURNAL, SEPTEMBER 19G3 

and "Kineplex" Data Transmission Systems, Proc. I.R.E., 47, Feb., 1959, 
pp. 333-334. 

4. Sunde, E. D., Ideal Pulses Transmitted by AM and FM, B. S.T.J. , 38, Nov., 

1959, pp. 1357-1426. 

5. Nyquist, H., Certain Topics in Telegraph Transmission Theory, Trans. 

A.I.E.E., 47, April, 1928, pp. 617-644. 

6. Knopp, K., Theory and Applications of Infinite Series, Blackie and Son, Ltd., 

London, 1928, p. 269. 

7. Meyerhoff, A. A. and Mazer, W. M., Optimum Binary FM Reception Using 

Discriminator Detection and I-F Shaping, RCA Review, 22, Dec, 1961, pp. 
698-728. 

8. Smith, E. F., Attainable Error Probabilities in Demodulation of Random 

Binary PCM/FM Waveforms, I.R.E. Trans, on Space Elec. and Telemetry, 
SET-8, Dec, 1962, pp. 290-297. 

9. Montgomery, G. F., A Comparison of Amplitude and Angle Modulation for 

Narrow-Band Communication of Binary-Coded Messages in Fluctuation 
Noise, Proc. I.R.E. , 42, Feb., 1954, pp. 447-454. 



